Architecture and communication protocol for haptic output devices

ABSTRACT

The embodiments are directed toward an architecture and communication protocol for controlling haptic output devices. According to the embodiments, a composite drive signal is generated that includes a first drive signal to be rendered by a first haptic output device, a second drive signal to be rendered by a second haptic output device, and a packet identifier. A controller includes the first haptic output device that is associated with a first user input element and the second haptic output device associated with a second user input element. The composite drive signal is transmitted to controller, and the execution order of the first and second drive signals is determined based on the packet identifier.

PRIORITY APPLICATION

This application claims the benefits of U.S. Provisional PatentApplication No. 62/096,251, filed on Dec. 23, 2014, which isincorporated herein by reference in its entirety.

FIELD OF INVENTION

The embodiments are generally directed to electronic devices, and moreparticularly, to electronic devices that produce haptic effects.

BACKGROUND

Video games and video game systems have become extremely popular. Videogame devices or controllers typically use visual and auditory cues toprovide feedback to a user. In some interface devices, kinestheticfeedback (e.g., active and resistive force feedback) and/or tactilefeedback (e.g., vibration, texture, temperature variation, and the like)may be provided to the user. In general, such feedback is collectivelyknown as “haptic feedback” or “haptic effects.” Haptic feedback providescues that enhance and simplify a user's interaction with a video gamecontroller, or other electronic device. For example, haptic effects mayprovide cues to users of video game controllers or other electronicdevices to alert the user to specific events, or provide realisticfeedback to create greater sensory immersion within a simulated orvirtual environment.

Other devices in which a user interacts with a user input element tocause an action also may benefit from haptic feedback or haptic effects.For example, such devices may include medical devices, automotivecontrols, remote controls, and other similar devices.

SUMMARY OF THE INVENTION

Embodiments of the present invention are directed toward electronicdevices configured to produce haptic effects that substantially improveupon the related art.

Features and advantages of the embodiments are set forth in thedescription which follows, or will be apparent from the description, ormay be learned by practice of the invention.

In one example embodiment, functionality for controlling haptic outputdevices according to an architecture and communication protocol isprovided. A composite drive signal is generated that includes a firstdrive signal to be rendered by a first haptic output device, a seconddrive signal to be rendered by a second haptic output device, and apacket identifier. A controller includes the first haptic output devicethat is associated with a first user input element and the second hapticoutput device associated with a second user input element. The compositedrive signal is transmitted to the controller, and the execution orderof the first and second drive signals is determined based on the packetidentifier.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot intended to limit the invention to the described examples.

BRIEF DESCRIPTION OF THE DRAWINGS

Further embodiments, details, advantages, and modifications will becomeapparent from the following detailed description of the preferredembodiments, which is to be taken in conjunction with the accompanyingdrawings.

FIG. 1 illustrates a block diagram of a system according to an exampleembodiment of the present invention.

FIG. 2 is a simplified block diagram illustrating a system for providinga haptic effect according to an example embodiment of the presentinvention.

FIG. 3 illustrates a block diagram of a haptic effect software stackaccording to an example embodiment of the present invention.

FIG. 4 is a simplified block diagram illustrating a system forcontrolling haptic communications according to an example embodiment ofthe present invention.

FIG. 5 illustrates a flow diagram of functionality for driving aplurality of haptic output devices according to an example embodiment ofthe present invention.

FIG. 6 illustrates a flow diagram of functionality for mapping positionsof a user input element with expected input signals according to anexample embodiment of the present invention.

FIG. 7 illustrates an example haptic stream report sent from a hostdevice to a controller according to an example embodiment to the presentinvention.

FIG. 8 illustrates an example haptic stream report sent from acontroller to a host device according to an example embodiment to thepresent invention.

FIG. 9 illustrates an example capability report sent from a controllerto a host device according to an example embodiment to the presentinvention.

FIG. 10 illustrates an example status request report sent from a hostdevice to a controller according to an example embodiment to the presentinvention.

FIG. 11 illustrates an example configuration change report sent from ahost device to a controller according to an example embodiment to thepresent invention.

FIG. 12 illustrates a functional block diagram of a controller suitablefor use with the embodiments of the present invention.

FIGS. 13A and 13B illustrate different views of a controller suitablefor use with the embodiments of the present invention.

DETAILED DESCRIPTION:

Example embodiments are directed toward functionality for controllinghaptic output devices according to an architecture and communicationprotocol. A composite drive signal is generated that includes a firstdrive signal to be rendered by a first haptic output device, a seconddrive signal to be rendered by a second haptic output device, and apacket identifier. A controller includes the first haptic output devicethat is associated with a first user input element and the second hapticoutput device associated with a second user input element. The compositedrive signal is transmitted to the controller, and the execution orderof the first and second drive signals is determined based on the packetidentifier.

In the various embodiments, a variety of user interfaces and methods forusing a device are described. In some embodiments, the device is aportable electronic device (e.g., a game controller, console, mobilephone, smartphone, tablet, etc.). It should be understood, however, thatthe user interfaces and associated methods may be applied to numerousother devices, such as personal computers, medical devices, laptops, andthe like that may include one or more other physical user-interfacedevices, such as a keyboard, mouse, trackball and the like.

FIG. 1 illustrates a block diagram of a system 100 according to anexample embodiment of the present invention.

System 100 may include a communication device 110 configured to transmitand/or receive data from remote sources. Communication device 110 mayenable connectivity between a processor 120 and other devices byencoding data to be sent from processor 120 to another device over anetwork (not shown) and decoding data received from another system overthe network for processor 120.

For example, communication device 110 may include a network interfacecard that is configured to provide wireless network communications. Avariety of wireless communication techniques may be used includinginfrared, radio, Bluetooth, Wi-Fi, and/or cellular communications.Alternatively, communication device 110 may be configured to providewired network connection(s), such as an Ethernet connection.

Processor 120 may comprise one or more general or specific purposeprocessors to perform computation and control functions of system 100.Processor 120 may include a single integrated circuit, such as amicro-processing device, or may include multiple integrated circuitdevices and/or circuit boards working in cooperation to accomplish thefunctions of processor 120. In addition, processor 120 may executecomputer programs, such as an operating system 141, a communicationmodule 142, and other applications 143, stored within memory 140.

System 100 may include memory 140 for storing information andinstructions for execution by processor 120. Memory 140 may containvarious components for retrieving, presenting, modifying, and storingdata. For example, memory 140 may store software modules that providefunctionality when executed by processor 120. The modules may includeoperating system 141 that provides operating system functionality forsystem 100. The modules may further include communication module 142that is configured to control packet based communications between system100 and controller 150. In some instances, the packet basedcommunications are relayed between system 100 and controller 150 overone or more USB (“Universal Serial Bus”) channels. Communication module142 also interfaces with a variety of other applications 143, and relaysthe haptic instructions originating from other applications 143 usingone or more standard and/or proprietary communication protocols.Although USB channels are described herein as an example, theembodiments of the invention may be readily applied to othercommunication protocols, such as Bluetooth. Other applications 143within system 100 may include additional functionality, such asperipheral firmware configured to provide control functionality for aperipheral device, such as controller 150 (e.g., a gamepad, wearabledevice, etc.).

Non-transitory memory 140 may include a variety of computer-readablemedia that may be accessed by processor 120. In the various embodiments,memory 140 may include volatile and nonvolatile media, removable andnon-removable media. For example, memory 140 may include any combinationof random access memory (“RAM”), dynamic RAM (“DRAM”), static RAM(“SRAM”), read only memory (“ROM”), flash memory, cache memory, and/orany other type of non-transitory computer-readable media. Alternatively,or additionally, memory 140 may include one or more network or cloudaccessible storage media.

Although shown as a single system, the functionality of system 100 maybe implemented as a distributed system. For example, memory 140 andprocessor 120 may be distributed across multiple different computersthat collectively comprise system 100. In one embodiment, system 100 maybe part of a device (e.g., personal computer, console, video gameconsole, etc.), and system 100 provides haptic effect functionality forthe device. In another embodiment, system 100 may be separate from thedevice, and may remotely provide the aforementioned functionality forthe device.

System 100 may be operably connected to controller 150. Controller 150may be a peripheral device configured to provide input to the system100. Controller 150 may be operably connected to system 100 using eithera wireless connection or a wired connection. Controller 150 also mayinclude a local processor configured to communicate with system 100using either a wireless connection or a wired connection. Alternatively,controller 150 may be configured to not include a local processor, andall input signals and/or output signals associated with controller 150may be processed by the components of system 100. In embodiments inwhich controller 150 has a local processor, additional functionality,such as communication modules and peripheral firmware configured toprovide control functionality may reside within controller 150. Here,the clock of the host device and the clock of the peripheral device maybe synchronized.

Controller 150 may further include one or more digital buttons, one ormore analog buttons, one or more bumpers, one or more directional pads,one or more analog or digital sticks, one or more driving wheels, and/orone or more user input elements that can be interacted with by a user,and that can provide input to system 100. Controller 150 may alsoinclude one or more analog or digital trigger buttons (or “triggers”)that can further be interacted with by the user, and that can furtherprovide input to system 100. As is described below in greater detail,controller 150 can further include a motor, or another type of actuatoror haptic output device, configured to exert a bi-directional push/pullforce on at least one trigger of controller 150.

Controller 150 can also include one or more actuators, or other types ofhaptic output devices. The local processor of controller 150, orprocessor 120 in embodiments where controller 150 does not include alocal processor, may transmit a haptic signal associated with a hapticeffect to at least one actuator of controller 150. The actuator, inturn, outputs haptic effects such as vibrotactile haptic effects,kinesthetic haptic effects, or deformation haptic effects, in responseto the haptic signal. The haptic effects can be experienced at a userinput element (e.g., a digital button, analog button, bumper,directional pad, analog or digital stick, driving wheel, or trigger) ofcontroller 150. Alternatively, the haptic effects can be experienced atan outer surface of controller 150.

An actuator is an example of a haptic output device, where a hapticoutput device is a device configured to output haptic effects, such asvibrotactile haptic effects, electrostatic friction haptic effects,temperature variation, and/or deformation haptic effects, in response toa drive signal. In alternate embodiments, the one or more actuatorswithin controller 150 can be replaced by some other type of hapticoutput device. The haptic output device may be, for example, an electricmotor, an electro-magnetic actuator, a voice coil, a shape memory alloy,an electro-active polymer, a solenoid, an eccentric rotating mass motor(“ERM”), a harmonic ERM motor (“HERM”), a linear resonant actuator(“LRA”), a piezoelectric actuator, a high bandwidth actuator, anelectroactive polymer (“EAP”) actuator, an electrostatic frictiondisplay, or an ultrasonic vibration generator. In some instances, thehaptic output device may include haptic output drive circuit. In someembodiments, the haptic output device may be unidirectional orbidirectional.

Controller 150 may further include one or more speakers. The localprocessor of controller 150, or processor 120 in embodiments wherecontroller 150 does not include a local processor, may transmit an audiosignal to at least one speaker of controller 150, which in turn outputsaudio effects. The speaker may be, for example, a dynamic loudspeaker,an electrodynamic loudspeaker, a piezoelectric loudspeaker, amagnetostrictive loudspeaker, an electrostatic loudspeaker, a ribbon andplanar magnetic loudspeaker, a bending wave loudspeaker, a flat panelloudspeaker, a heil air motion transducer, a plasma arc speaker, and adigital loudspeaker.

Controller 150 can further include one or more sensors. A sensor may beconfigured to detect a form of energy, or other physical property, suchas, but not limited to, sound, movement, acceleration, bio signals,distance, flow, force/pressure/strain/bend, humidity, linear position,orientation/inclination, radio frequency, rotary position, rotaryvelocity, manipulation of a switch, temperature, vibration, or visiblelight intensity. The sensor may further be configured to convert thedetected energy, or other physical property, into an electrical signal,or any signal that represents virtual sensor information, and controller150 can send the converted signal to the local processor of controller150, or processor 120 in embodiments where controller 150 does notinclude a local processor.

FIG. 2 is a simplified block diagram illustrating a system 200 forproviding a haptic effect according to an example embodiment of thepresent invention.

As shown in FIG. 2, system 200 includes one or more applications 210.Each of applications 210 may generate a variety of instructions 215.Instructions 215 are supplied to a communication driver 220. Uponreceiving instructions 215, communication driver 220 may generate one ormore haptic drive signals 225 based on instructions 215. In someconfigurations, haptic drive signals 225 may be configured according toone or more report formats. Haptic drive signals 225 are thentransmitted to a controller 230.

At controller 230, haptic drive signals 225 may be executed by one ormore haptic output devices 232. By executing haptic drive signals 225,haptic output devices 232 render the haptic effects to an end user.

FIG. 3 illustrates a block diagram of a haptic effect software stack 300according to an example embodiment of the present invention. As shown inFIG. 3, software stack 300 includes device modules 310, peripheralfirmware modules 320, controller modules 330, drive modules 340, andrumble drive modules 350. Haptic effect software stack 300 isimplemented on a system, such as system 100 of FIG. 1.

Device modules 310 may include a variety of modules such as inputmanagement code 311, peripheral input application programming interface(“API”) 312, rumble API 313, haptic effect API 314, directplayback/crossover 315, trigger engine 316, spatialization engine 317,and encoder 318.

Input management code 311 may include a set of computer-readableinstructions that manage input provided by controller 330 in the contextof a game application, or other type of application, executed within adevice.

Peripheral input API 312 may include a set of computer-readablefunctions or routines that enable game input management code 311 tointeract with peripheral firmware 320 in order to receive and manageinput provided by controller 330.

Rumble API 313 may include a set of computer-readable functions orroutines that enable input management code 311 to interact withperipheral firmware 320 in order to transmit rumble instructions to oneor more rumble motors or rumble actuators of controller 330 (e.g.,rumble motors L and R of FIG. 3). In addition, a rumble instruction maycause a rumble motor or rumble actuator of controller 330 to produce ageneral or rumble haptic effect.

Haptic effect API 314 (identified in FIG. 3 as “API”) may include a setof computer-readable functions or routines that are accessible to inputmanagement code 311, and that enable input management code 311 tointeract with peripheral firmware 320 in order to transmit hapticinstructions to controller 330. In addition, a haptic instruction maycause one or more targeted motors or targeted actuators of controller330 to produce a haptic effect at one or more user input elements ofcontroller 330.

Haptic effect API 314 also may store one or more haptic effectdefinitions. A haptic effect definition is a data structure thatincludes haptic data, such as a haptic signal, that is pre-defined andthat can be stored within a storage, such as a haptic file or hapticstream, and that can be sent to one or more rumble motors, rumbleactuators, targeted motors, or targeted actuators, to produce a hapticeffect at a component, or user input element, of controller 330. Thehaptic data can include one or more attributes of the correspondinghaptic effect, where the attributes can be stored as parameters. Exampleparameters of a haptic effect definition may include an amplitudeparameter, a frequency parameter, a waveform parameter, an envelopeparameter, a magnitude (or strength) parameter, and a durationparameter.

Haptic effect API 314 may enable game input management code 311 tointeract with direct playback/crossover 315, trigger engine 316, andspatialization engine 317, and may further manage directplayback/crossover 315, trigger engine 316, and spatialization engine317 according to requests invoked by game input management code 311.Further, haptic effect API 314 may store data used for communicationwith peripheral firmware 320, and used for generation of one or morehaptic effects.

Direct playback/crossover 315 may receive haptic data as input, producehaptic data as output, and transmit haptic data to one or more targetedmotors, or targeted actuators, of controller 330 (e.g., motors L and Rof FIG. 3). In some embodiments, direct playback/crossover 315 mayoutput the input haptic data directly, without modifying a format of theinput haptic data. This results in an “as-is” playback of the inputhaptic data. In other embodiments, direct playback/crossover 315 mayconvert the haptic data that is input from a first format to a secondformat, and can further output the converted haptic data. Depending onthe type of playback, direct playback/crossover 315 may optionally use aprogrammable crossover to convert the haptic data. By converting thehaptic data, device modules may deconstruct the haptic effect andplayback the haptic effect at multiple actuators.

The format of the haptic data may be a haptic elementary stream (“HES”)format. A HES format is a file or data format for representing hapticdata that may be streamed to a device. The haptic data can berepresented in a manner that is identical or similar to how uncompressedsound is represented, although the haptic data can be encrypted withinthe HES format.

Trigger engine 316 may receive haptic data, such as a haptic effectdefinition, and may modify the haptic data based on user input data,such as trigger data 323. Trigger data is data that includes one or moreparameters that indicate a position and/or range of one or more triggersof controller 330 (e.g., triggers L and R of FIG. 3). Trigger engine 316may further transmit haptic instructions to controller 330. For example,trigger engine 316 may transmit haptic instructions to a variety ofuser-input elements of controller 330. As previously described, a hapticinstruction may cause one or more targeted motors or targeted actuatorsof controller 330 to produce a haptic effect at one or more user-inputelements of controller 330.

Spatialization engine 317 may receive haptic data and may modify thehaptic data based on spatialization data. Spatialization data mayinclude data that indicates a desired direction and/or flow of a hapticeffect, such as an ordering of haptic effects on respective user inputelements. In certain embodiments, spatialization engine 317 may receivespatialization data that includes a direction and/or flow from inputmanagement code 311.

Spatialization engine 317 may modify the haptic data so that a hapticeffect, such as a trigger haptic effect, is scaled for one or morerumble motors, or rumble actuators, of controller 330 (e.g., rumblemotors L and R of FIG. 3), and that the haptic effect is also scaled forone or more targeted motors, or targeted actuators, of controller 330(e.g., motors L and R, as illustrated in FIG. 3). In other words,spatialization engine 317 may modify the haptic data that is sent toeach motor or actuator, and thus, modify the haptic effect that isexperienced at each motor or actuator, in order to convey a sense ofdirection and flow of an overall haptic effect. For example, in order toemphasize a haptic effect experienced at a motor or actuator,spatialization engine 317 may scale one or more portions of the hapticeffect. For example, spatialization engine 317 may scale haptic datathat is sent to the motor or actuator that causes the haptic effect tobe experienced, causing the haptic effect to be more pronounced (e.g.,increased magnitude, duration, etc.). Additionally, spatializationengine 317 may scale haptic data that is sent to other motors oractuators, causing other haptic effects that are experienced at thosemotors or actuators to be less pronounced (e.g., decreased magnitude,duration, etc.). In some embodiments, spatialization engine 317 maymodify the haptic data in real-time or substantially in real time.Further, in some embodiments, spatialization engine 317 may havenon-linear relationships between inputs and motor, or actuator, outputsin order to exaggerate an overall haptic effect.

Encoder 318 encodes haptic data received from direct playback/crossover315, trigger engine 316, and/or spatialization engine 317 into a format.In one embodiment, the format may be an HES format. Encoder 318 maytransmit the encoded haptic data to peripheral firmware 320.

Peripheral firmware 320 is firmware for one or more peripheral devices(e.g., controllers). Peripheral firmware 320 may include a variety ofmodules such as decoder and crossover 321, trigger control 322, triggerdata 323, other functions 324, and rumble control 325.

Decoder and crossover 321 may receive the encoded haptic data fromencoder 318 and decodes the encoded haptic data. In some embodiments,decoder and crossover 321 computes a programmable crossover in order todecode the encoded haptic data. Decoder and crossover 321 may computethe programmable crossover in real-time.

Trigger control 322 is a low-level control API for one or more targetedmotors or targeted actuators of controller 330 (e.g., motors L and R ofFIG. 3). Trigger control 322 may receive a trigger instruction and mayconvert the trigger instruction into a low-level trigger instruction fora specified targeted motor or targeted actuator of controller 330, andmay transmit the low-level trigger instruction to the specified targetedmotor or targeted actuator of controller 330. The low-level triggerinstruction may cause the specified targeted motor or targeted actuatorto produce a trigger haptic effect at a specified trigger of controller330.

Trigger data 323, as previously described, is data that includes one ormore parameters that indicate a position and/or range of one or moretriggers of controller 330 (e.g., triggers L and R of FIG. 3). Triggerdata 323 may be received from controller 330 by peripheral firmware 320.Peripheral firmware 320 may further store trigger data 323, and mayfurther transmit trigger data 323 to device modules 310.

Other gamepad functions 324 may be functions of controller 330 managedby peripheral firmware 320. Such functions may include such functions aswired/wireless communications, input reporting, protocol implementation,power management, etc.

Rumble control 325 is a low-level control API for one or more rumblemotors or rumble actuators of controller 330 (e.g., rumble motors L andR of FIG. 3). Rumble control 325 may receive a rumble instruction, mayconvert the rumble instruction into a low-level rumble instruction for aspecified rumble motor or rumble actuator of controller 330, and maytransmit the low-level trigger instruction to the specified rumble motoror rumble actuator of controller 330.

Communication driver module 326 is a firmware module that is configuredto control packet based communications between device modules 310 andcontroller 330. For example, the packet based communications may berelayed between device modules 310 and controller 330 over one or moreUSB channels. Upon receiving instructions from device modules 310,communication driver module 326 may generate one or more haptic drivesignals. In some configurations, the haptic drive signals are generatedaccording to one or more report formats.

Communication driver module 326 may include multiple sections offirmware that may be hardware dependent or independent. In someinstances, sections of firmware that are hardware independent may beseparated from the sections that are hardware dependent. Here, hardwareindependent firmware may interact with the hardware dependent firmwareby using functional pointers.

Controller 330 may include triggers L and R. Controller 330 may furtherinclude gear boxes L and R and motors L and R. Motor L and gearbox L areoperably coupled to trigger L within controller 330. Likewise, motor Rand gearbox R are operably coupled to trigger R within controller 330.When motor L receives a trigger instruction, motor L and gearbox L maycollectively cause a trigger haptic effect to be experienced at triggerL. Likewise, when motor R receives a trigger instruction, motor R andgearbox R may collectively cause a trigger haptic effect to beexperienced at trigger R. Peripheral firmware 320 may send triggerinstructions to motors L and R of controller 330 using drive electronics340.

Controller 330 may further include potentiometers L and R. PotentiometerL may detect a position and/or range of trigger L, and may further sendthe detected position and/or range of trigger L to peripheral firmware320 as trigger data. Likewise, potentiometer R may detect a positionand/or range of trigger R, and may further send the detected positionand/or range of trigger R to peripheral firmware 320 as trigger data.

Controller 330 may further include rumble motors L and R. When rumblemotor L receives a rumble instruction, rumble motor L causes a hapticeffect to be experienced along a left component of controller 330.Likewise, when rumble motor R receives a rumble instruction, rumblemotor R causes a haptic effect to be experienced along a right componentof controller 330. Peripheral firmware 320 may send rumble instructionsto rumble motors L and R using rumble drive electronics 350.

FIG. 4 is a simplified block diagram illustrating a system 400 forcontrolling haptic communications according to an example embodiment ofthe present invention.

As an input to system 400, one or more applications 410 may generate avariety of haptic instructions. In some instances, the hapticinstructions may be retrieved from a haptic effect library 415. Thehaptic instructions may be parsed, mixed, and converted to packetizedhaptic drive signals suitable for transmission on a USB compliantchannel by the various components of system 400.

A haptic engine 420 is a higher level API (“Application ProgrammingInterface”) that uses lower level API functions to manage the hapticeffect instructions originating from applications 410. For example,haptic engine 420 may be configured to load, start, stop, modify, and/orrender the playback of the haptic effects.

Haptic engine 420 interfaces with haptic effect parser 430. As its nameimplies, haptic effect parser 430 is configured to parse haptic effectinstructions into segments defining their operational characteristics.In some instances, haptic effect parser 430 also may retrieve additionaloperational characteristics from memory (e.g., haptic effect library415). Alternatively, or additionally, haptic effect parser 430 includesan API to load the haptic effects in memory, verify haptic effectformats, and retrieve a variety of information relating to the hapticeffects, such as packet size, effect duration, and other haptic effectdata.

Haptic engine 420 also interfaces with a haptic mixer 440. At hapticmixer 440, haptic instructions intended for different haptic outputdevices may be combined into a single haptic instruction set. Byimplementing haptic mixer 440, system 400 may simultaneously playbackmultiple haptic effects at different haptic output devices. Here, startand/or stop times for the various haptic effects may be adjusted and/ordetermined. In addition, haptic mixer 440 may manage the contents of amixer buffer (not shown).

Haptic engine 420 further interfaces with a haptic device handler 450.Haptic device handler 450 initiates and manages communication with thehaptic output devices. For example, haptic device handler 450 isconfigured to retrieve individual haptic output device handlers (e.g.,identifiers). Accordingly, haptic device handler 450 may render hapticeffects at the various haptic output devices of system 400. In someinstances, haptic device handler 450 also initializes several statemachine structures utilized to render the haptic effects.

Further, haptic device handler 450 interfaces with the USB communicationlayers, including haptic report handler 460 and USB HID (“HumanInterface Device”) library 470. Haptic report handler 460 packages thehaptic instructions originating from applications 410 into packetizeddata suitable for transmission on a USB channel. And, USB HID library470 stores a variety of functions used to facilitate USB communications.For example, USB HID library 470 stores functions to encode and decodethe haptic instructions. In another example, USB HID library 470 storesUSB descriptors and functions to handle the USB communication.

By utilizing the various components of system 400, including components420-470, USB compliant communications between the controller (such ascontroller 230 of FIG. 2) and application 410 may be achieved. Betweenthe controller and application 410, additional layers such as gamepadfirmware 480 and controller input reader 490 may be used. For example,controller input reader 490 reports the state of the user input elementsof the controller to applications 410.

FIG. 5 illustrates a flow diagram of functionality for driving aplurality of haptic output devices according to an example embodiment ofthe present invention. In some configurations, the functionality of theflow diagram of FIG. 5 (and FIG. 6 below) may be implemented by softwarestored in memory or other computer readable or tangible media, andexecuted by a processor. In other embodiments, the functionality may beperformed by hardware (e.g., through the use of an application specificintegrated circuit (“ASIC”), a programmable gate array (“PGA”), a fieldprogrammable gate array (“FPGA”), etc.), or any combination of hardwareand software.

At the outset, functionality 500 generates a composite drive signal thatincludes a first drive signal to be rendered by a first haptic outputdevice, a second drive signal to be rendered by a second haptic outputdevice, and a packet identifier, at 510. Here, the first haptic outputdevice may be associated with a first user input element and the secondhaptic output device may be associated with a second user input element.Next, at 520, the composite drive signal is transmitted to a controllerhaving the first and second haptic output devices. Lastly, the executionorder of the first and second drive signals is determined based on thepacket identifier, 530.

FIG. 6 illustrates a flow diagram of functionality 600 for mappingpositions of a user input element with expected input signals accordingto an example embodiment of the present invention.

At the outset, the user input elements of the controller may beinitialized, at 610. Here, functionality 600 may initially set positionand range information for the user input elements. In some instances,these values may be calculated based on the movement of the user inputdevice from the maximum out position to the grounding position.

Next, functionality 600 determines and stores profiles for the userinput elements, at 620. The determined profiles may map each position ofthe user input device to an analog to digital conversion (“ADC”) value.For example, the determined profiles of 620 may map each position of theuser input device to an ADC value between 0 and 255.

The determined profiles may utilize either an increasing or a decreasingprofile. For example, an increasing profile will produce a value [0,255]when the position of the user input value is read from an 8 bit ADCdata. Similarly, a decreasing profile will produce a value [255,0] whenread from the 8 bit ADC data.

Subsequently, at 630, functionality 600 determines and stores anexpected input signal for each position of the user input device. Insome instances, ranges of user input values may be associated withexpected input signals.

In some instances, the resting position of the user input elements mayvary at different times. For example, after use of the various userinput devices, some of the user input devices may not return to the sameresting position when the user interaction is removed. In suchinstances, functionality 600 may adjust the determined profile andexpected user input values for such user input elements, at 640.Accordingly, the changed resting position(s) may be accounted for whilemonitoring the position of the user input elements.

FIG. 7 illustrates an example haptic stream report 700 sent from a hostdevice to a controller according to an example embodiment to the presentinvention. Within haptic stream report 700, a report identifier 710 maybe associated with a timer identifier 720, and haptic drive packets 730.Haptic drive packets may include a predetermined number of bytes (e.g.,25 bytes) allocated to each of the haptic output devices (e.g., leftrumble, right rumble, left trigger, right trigger, etc.). The overalllength of haptic stream report 700 may be indicated by number of bytesidentifier 740.

Within haptic drive packets 730, a force value may be modified and/orapplied to each haptic output device when the haptic stream report isupdated (e.g., periodically every 5ms). For example, if the host devicereceives a request to play a new haptic effect, it may retrieve hapticeffect data from memory, and populate haptic stream report 700 withforce values to be applied on multiple haptic output devices for anupcoming 70ms (e.g., 14 force values per actuator yields 14*5ms). Here,timer identifier 720 such as start tick value, at byte offset 2, may beused to synchronize haptic instructions by ensuring their storage at thecorrect locations within the drive buffers of the haptic output devices.

In the various configurations, the semantics and the range of forcevalues may depend on the haptic output device and its directionality.For example, two types of haptic output devices include trigger andrumble actuators. Trigger actuators are typically bidirectional and areconfigured to respond to push and pull motions. By contrast, rumbleactuators may be bidirectional or unidirectional. The range of forcevalues for both rumble and trigger actuators may be the same (e.g., 0x00and 0xFF). However, the haptic output devices may utilize range of forcevalues differently based on their respective characteristics.

For the trigger actuators example, the force values in the range [0,127]may move the trigger actuators in a pull direction. Here, the strengthof the pull may decrease as the force value increases. The force valueof 128 may not push or pull the trigger actuators. And, the force valuesin the range [129,255] may move the trigger actuators in a pushdirection. Here, the strength of the push increases as the force valueincreases.

In another example, for unidirectional rumble actuators, the forcevalues in the range [0,127] may be ignored. The force value of 128 maynot move the unidirectional rumble actuator, and the force value in therange [129,255] may move the unidirectional rumble actuator in theclockwise direction with speed increasing as the force value increases.

In yet another example, for bidirectional rumble actuators, the forcevalues in the range [0,127] may move the actuators in thecounter-clockwise direction. The force value of 128 may not move theactuator. The force values in the range [129,255] may move the actuatorsin the clockwise direction with speed increasing as the force valueincreases.

Thus, haptic stream report 700 provides an example composite drivesignal. In this example configuration, drive signals for a plurality ofhaptic output devices may be simultaneously supplied to the controllerwithin one reporting structure (e.g., 700). In turn, the controller maydetermine the execution (or execution order) for the plurality of hapticoutput devices based on the various fields within haptic stream report700, such report identifier 710, timer identifier 720, and haptic drivepackets 730.

FIG. 8 illustrates an example haptic stream report 800 sent from acontroller to a host device according to an example embodiment to thepresent invention. Haptic stream report 800 may communicate informationabout the positions of the user input elements associated with thehaptic output devices. For example, haptic stream report 800 may storedata for six potentiometer values, of 1 byte each, to convey the triggerand joystick positions. Example haptic stream report 800 can store andreport up to sixteen buttons allocating 1 bit for every button state.

FIG. 9 illustrates an example capability report 900 sent from acontroller to a host device according to an example embodiment to thepresent invention. Capability report 900 may communicate controllercapabilities, such as the number of haptic output devices, the hapticdevice update rates, and the buffer identifiers allocated to each hapticoutput device. In addition, capability report 900 may further identifycommunication protocols and/or firmware supported by the controller.

FIG. 10 illustrates an example status request report 1000 sent from ahost device to a controller according to an example embodiment to thepresent invention. For example, the host device may check the status ofthe most recent USB packet or command. In response to status requestreport 1000, the controller may report that the command is valid,invalid, unsupported, and the like.

FIG. 11 illustrates an example configuration change report 1100 sentfrom a host device to a controller according to an example embodiment tothe present invention. Using configuration change report, the controllermay request changes to the firmware settings of the host device. Forexample, configuration change report 1100 may be used to change thehaptic output device update rate.

Each of the reports 800-1100 of FIGS. 8-11 illustrates an additionalexample composite signal. In each of these examples, the positions,capabilities, status, and configuration of multiple user input elementsand haptic output devices may be simultaneously communicated between thehost device and the controller. In each of reports 800-1100, variousidentifiers, such as report, command, module, parameter, and otheridentifiers may be used.

FIG. 12 illustrates a functional block diagram of a controller 1200suitable for use with the embodiments of the present invention.

As illustrated in FIG. 12, controller 1200 may include one or more of avariety of user input elements. A user input element may refer to anyinterface device manipulated by the user to interact with host computer1204. Example user input elements include analog or digital joy stick1210, button 1214, trigger 1218, and the like. As understood by one ofordinary skill in the art, one or more of each user input element may beincluded on controller 1200. For example, the present description oftrigger 1218 does not limit controller 1200 to a single trigger.Similarly, those skilled in the art understand that multiple analog ordigital sticks, buttons, and other user input elements may be used.

Controller 1200 may include local processor 1208. Local processor 1208may exchange commands and data with host computer 1204 via connection1205. Connection 1205 may be a wired or wireless connection using one ormore communication protocols known to those skilled in the art. In someinstances, controller 1200 may be alternatively configured to notinclude local processor 1208. Here, input/output signals from controller1200 may be handled and processed directly by host computer 1204. Hostcomputer 1204 may be a gaming device console and display device 1206 maybe screen which is operably coupled to the gaming device console. Insome instances, host computer 1204 and display device 1206 may becombined into a single device.

Controller 1200 may include targeted actuators 1212, 1216, 1220 (e.g.,motors) to directly drive each of the user input elements thereof aswell as one or more general or rumble actuators 1222, 1224 operablycoupled to housing 1202 in a location where a hand of the user isgenerally located. More particularly, analog or digital stick 1210includes a targeted actuator or motor 1212 operably coupled thereto,button 1214 includes a targeted actuator or motor 1216 operably coupledthereto, and trigger 1218 includes a targeted actuator or motor 1220operably coupled thereto. In addition to a plurality of targetedactuators, controller 1200 includes a position sensor operably coupledto each of the user input elements thereof. More particularly, analog ordigital stick 1210 includes a position sensor 1211 operably coupledthereto, button 1214 includes a position sensor 1215 operably coupledthereto, and trigger 1218 includes a position sensor 1219 operablycoupled thereto. Local processor 1208 is operably coupled to targetedactuators 1212, 1216, 1220 as well as position sensors 1211, 1215, 1219of analog or digital stick 1210, button 1214, and trigger 1218,respectively. In response to signals received from position sensors1211, 1215, 1219, local processor 1208 instructs targeted actuators1212, 1216, 1220 to provide directed or targeted kinesthetic effectsdirectly to analog or digital stick 1210, button 1214, and trigger 1218,respectively. Such targeted kinesthetic effects are discernible ordistinguishable from general or rumble haptic effects produced bygeneral actuators 1222, 1224 along the entire body of the controller.The collective haptic effects provide the user with a greater sense ofimmersion to the game as multiple modalities are being simultaneouslyengaged (e.g., video, audio, and haptics).

FIGS. 13A and 13B illustrate different views of a controller 1300suitable for use with the embodiments of the present invention. As shownin FIGS. 13A and FIG. 13B, controller 1300 may include a variety ofcomponents such as housing 1302, analog or digital joy stick 1310,button(s) 1314, trigger 1318, and rumble actuators 1322 and 1324.

Housing 1302 is shaped to easily accommodate user gripping of controller900. Controller 1300 is an example embodiment of a controller, and theembodiments of the invention may be readily applied to other controllershapes.

Accordingly, the embodiments of the present invention provide animproved architecture and communication protocol for the haptic outputdevices. By implementing the various embodiments, drive signals formultiple haptic output devices may be simultaneously communicated andexecuted.

One having ordinary skill in the art will readily understand that theinvention as discussed above may be practiced with steps in a differentorder, and/or with elements in configurations which are different thanthose which are disclosed. Therefore, although the invention has beendescribed based upon these preferred embodiments, it would be apparentto those of skill in the art that certain modifications, variations, andalternative constructions would be apparent, while remaining within thespirit and scope of the invention. In order to determine the metes andbounds of the invention, therefore, reference should be made to theappended claims.

We claim:
 1. A method for driving a plurality of actuators, eachactuator being associated with one of a plurality of user inputelements, the method comprising: generating a composite drive signalpacket data structure that includes a first drive signal to be renderedby a first haptic output device, a second drive signal to be rendered bya second haptic output device, and a packet identifier, the first hapticoutput device associated with a first user input element and the secondhaptic output device associated with a second user input element;transmitting, to a controller, the composite drive signal packet datastructure; and determining an execution order of the first drive signaland the second drive signal at the controller according to the packetidentifier, wherein one or more haptic effects are rendered to a useraccording to the first drive signal and the second drive signal.
 2. Themethod according to claim 1, further comprising simultaneouslytransmitting the first and second drive signals.
 3. The method accordingto claim 1, further comprising synchronizing a first clock and a secondclock, the first clock being disposed at a host device, and the secondclock being disposed at a peripheral device.
 4. The method according toclaim 1, wherein the composite drive signal packet data structure isreceived on a USB compliant channel.
 5. The method according to claim 1,wherein the plurality of user input elements are disposed at aperipheral device comprising a controller or a gamepad.
 6. The methodaccording to claim 1, wherein the first or second user input element isa trigger element of a peripheral device.
 7. A device comprising: aprocessor; and a memory storing a program for execution by theprocessor, the program including instructions for: generating acomposite drive signal packet data structure that includes a first drivesignal to be rendered by a first haptic output device, a second drivesignal to be rendered by a second haptic output device, and a packetidentifier, the first haptic output device associated with a first userinput element and the second haptic output device associated with asecond user input element; transmitting, to a controller, the compositedrive signal packet data structure; and determining an execution orderof the first drive signal and the second drive signal at the controlleraccording to the packet identifier, wherein one or more haptic effectsare rendered to a user according to the first drive signal and thesecond drive signal.
 8. The device according to claim 7, furthercomprising instructions for simultaneously transmitting the first andsecond drive signals.
 9. The device according to claim 7, furthercomprising instructions for synchronizing a first clock and a secondclock, the first clock being disposed at a host device, and the secondclock being disposed at a peripheral device.
 10. The device according toclaim 7, wherein the composite drive signal packet data structure isreceived on a USB compliant channel.
 11. The device according to claim7, wherein the first user input element and second user input elementare disposed at a peripheral device comprising a controller or agamepad.
 12. The device according to claim 7, wherein the first orsecond user input element is a trigger element of a peripheral device.13. A non-transitory computer readable storage medium storing a programconfigured to be executed by a processor, the program comprisinginstructions for: generating a composite drive signal packet datastructure that includes a first drive signal to be rendered by a firsthaptic output device, a second drive signal to be rendered by a secondhaptic output device, and a packet identifier, the first haptic outputdevice associated with a first user input element and the second hapticoutput device associated with a second user input element; transmitting,to a controller, the composite drive signal packet data structure; anddetermining an execution order of the first drive signal and the seconddrive signal at the controller according to the packet identifier,wherein one or more haptic effects are rendered to a user according tothe first drive signal and the second drive signal.
 14. Thenon-transitory computer readable storage medium of claim 13, furthercomprising instructions for simultaneously transmitting the first andsecond drive signals.
 15. The non-transitory computer readable storagemedium of claim 13, further comprising instructions for synchronizing afirst clock and a second clock, the first clock being disposed at a hostdevice, and the second clock being disposed at a peripheral device. 16.The non-transitory computer readable storage medium of claim 13, whereinthe composite drive signal packet data structure is received on a USBcompliant channel.
 17. The non-transitory computer readable storagemedium of claim 13, wherein the first and second user input elements aredisposed at a peripheral device comprising a controller or a gamepad.18. The non-transitory computer readable storage medium of claim 13,wherein the first or second user input element is a trigger element of aperipheral device.
 19. The method according to claim 1, wherein thecomposite drive signal packet data structure is determined by parsinghaptic effect instructions of the one or more haptic effects into firsthaptic instructions for the first drive signal and second hapticinstructions for the second drive signal; and mixing the first hapticinstructions and the second haptic instructions into a single hapticinstruction set.